Skip to content

Add Python integration #572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 28, 2024
Merged

Add Python integration #572

merged 2 commits into from
Sep 28, 2024

Conversation

jpsamaroo
Copy link
Member

@jpsamaroo jpsamaroo commented Sep 26, 2024

Heresy, I say! 🐍

This PR adds integration with Python, such that Dagger can be made to execute Python functions on Python arguments. This will pair with a new Python library called pydaggerjl, which provides a Pythonic interface to Dagger via JuliaCall. While only the basics (spawn, fetch, wait) are there now, we'll be adding more APIs to pydaggerjl for DArrays, Datadeps, and more.

While this PR is not strictly necessary (as calling Python functions already actually works, thanks to PythonCall), this formalizes our Python support by creating a single PythonProcessor per process, which matches the semantics of Python running with GIL limitations (so we only run one Python function at a time, regardless of how many Julia threads we have).

An expected question is: Why do this at all? Isn't Julia the answer to the two-language problem, so we shouldn't concern ourselves with other languages? Well, that could be fine if Python was a very niche language that few people use. Instead, Python dominates the landscape of software, together with other languages like C/C++ and R (expect to see PRs for those in the future!). Additionally, when giving presentations about Dagger to broad audiences, we inevitably have at least one question each time about how to use Dagger with language XYZ, and have never really had a good answer to this. Therefore, I feel that it's time we embrace the possibility of cross-language interop, and allow Dagger to gain adoption outside of the Julia ecosystem.

Example from Python:

import numpy as np
from pydaggerjl import daggerjl

# Create a Dagger DTask to sum the elements of an array
task = daggerjl.spawn(np.sum, np.array([1, 2, 3]))

# Fetch the result
result = daggerjl.fetch(task)

print(f"The sum is: {result}")

# Create two numpy arrays
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Element-wise sum of the two arrays
task = daggerjl.spawn(np.add, a, b)

# Fetch the result
result = daggerjl.fetch(task)

print(f"The element-wise sum is: {result}")

# Element-wise sum of last result with itself
task2 = daggerjl.spawn(np.add, task, task)

# Fetch the result
result2 = daggerjl.fetch(task2)

print(f"The element-wise sum of the last result with itself is: {result2}")

@jpsamaroo jpsamaroo merged commit 2e9b120 into master Sep 28, 2024
1 of 2 checks passed
@jpsamaroo jpsamaroo deleted the jps/python branch September 28, 2024 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant